Image processing apparatus, image processing method, program and semiconductor integrated circuit

ABSTRACT

An image processing apparatus includes a filter unit which filters image signals; a sampling unit which generates first digital image signals having a first resolution by sampling the filtered image signals at a predetermined sampling frequency; and a super-resolution unit which reconstructs a second digital image signal having a second resolution which is higher than the first resolution by performing super-resolution on the first digital image signals generated by the sampling unit, wherein the filter unit passes frequency components corresponding to or lower than the Nyquist frequency which is half the sampling frequency, and passes a part of frequency components within a range from the Nyquist frequency to the highest frequency which can be represented by the second resolution.

BACKGROUND OF THE INVENTION

1. Technical Field

The present invention relates to an image processing apparatus, an imageprocessing method, a program and a semiconductor integrated circuit forreconstructing a high-resolution digital image signal by performingsuper-resolution interpolation on image signals.

2. Background Art

Recently, methods for converting a sequence of low-resolution imagesinto a high-resolution image or a sequence of high-resolution imageshave attracted considerable interest among computer scientists and imageprocessing specialists. These methods are commonly referred to assuper-resolution, super-resolution interpolation or super-resolutionreconstruction. The basic idea behind super-resolution is to exploitmotions in low-resolution images at sub-pixel level in order toreconstruct image details that are not apparent from any one of theseimages by itself.

Super-resolution techniques are particularly interesting in the contextof image acquisition, since they provide an efficient method to improveimage resolution without employing costly high-performance imagingdevices.

FIG. 1A is a block diagram of a conventional image acquisition system.

As shown in FIG. 1A, a sampling unit 120 samples an input image 101 at apredetermined sampling frequency. The processing/recording unit 150processes it or records it onto a recording medium. In the case wherethe input image contains video frequencies which are higher than theNyquist frequency of the sampling unit 120, aliasing, that is, foldingnoise occurs in the sampled image. This can be avoided by ananti-aliasing filter (folding noise prevention filter) 110 as shown inFIG. 1B. The anti-aliasing filter 110 is a low-pass filter which removesvideo frequencies exceeding the Nyquist frequency before the sampling.Hence, the conventional image acquisition and reproduction apparatusshown in FIG. 1B outputs an image 190 free of folding noise.

Functions of a conventional anti-aliasing filter 110 are described withreference to FIGS. 2A to 2C and FIGS. 3A to 3E. For simplification, asignal is assumed to be a one-dimensional signal here. In each of thedrawings, the left-hand graph shows a signal in a spatial domain. Thehorizontal axis x shows one-dimensional spatial coordinate or a timeaxis, and the longitudinal axis shows luminance. Likewise, theright-hand graph shows a signal transformed (Fourier-transformed) into afrequency domain. The horizontal axis ω shows frequency (radian), thelongitudinal axis shows frequency strength, and ω_(N) shows the Nyquistfrequency.

FIG. 2A shows a rapidly varying video signal with an accordingly broadfrequency spectrum-F. Sampling this signal can be expressed as amultiplication with a Dirac comb g as represented schematically in FIG.2B. Note that the Fourier transform of a Dirac comb is also a Dirac combG. Since multiplication of two signals in the spatial domain correspondsto a convolution of the transformed signals in the frequency domain, thespectrum of the sampled signal F*G takes the form as indicated on theright-hand side of FIG. 2C. In other words, the spectrum F*G of thesampled signal (shown as solid lines) is the sum of the spectra (shownas dashed lines) transformed and replicated periodically.

As shown in FIG. 2C, the transformed and replicated spectra (shown asdashed lines) overlap with each other. The spectral power of the sampledsignal at a certain frequency is thus contaminated by contributions fromother frequencies that are a so-called alias to the certain frequency.In the spatial domain, aliasing artifacts become clear and apparentnoise such as Moiré patterns or jaggy which occurs along smooth edgeline portions.

In order to prevent aliasing, it is thus necessary to preventoverlapping of the spectra in the sampling. This can be achieved byband-limiting the initial signal by means of a low-pass filter h_(AA) asshown in FIG. 3B. The anti-aliasing filter 110 has characteristics ofthe low-pass filter h_(AA).

FIG. 3A represents the video signal f, and FIG. 3B represents thelow-pass filter h_(AA) used for band-limiting the signal by convolvingthe signal in the spatial domain or multiplying the signal in thefrequency domain. FIG. 3C shows the result f*h_(AA) of the low-passfiltering (shown as a solid line) in comparison to the video signal f(shown as a dashed line).

As explained above, sampling corresponds to multiplication of the signalwith a Dirac comb g in the spatial domain, and to a convolution with thecorresponding Dirac comb G in the frequency domain (cf. FIG. 3D). Sincethe spectrum of the signal has been band-limited, the transformed andreplicated spectra F*H_(AA) do no longer overlap with each other (cf.FIG. 3E), so that no aliasing occurs.

Next, conventional image acquisition systems with super-resolutioninterpolation are illustrated. FIG. 4A is a block diagram of theconfiguration of the conventional image acquisition system including thesampling unit 120, the processing/recording unit 150, and thesuper-resolution interpolation unit 160. The input image 101 is sent tothe sampling unit 120. The sampling unit 120 generates a digital imageby sampling the input image 101 at a predetermined sampling frequency.The processing/recording unit 150 outputs, as a low-resolution outputimage 191, the digital image at a sampling resolution of the originalimage. Otherwise, the low-resolution output image is outputted to thesuper-resolution interpolation unit 160. The super-resolutioninterpolation unit 160 outputs a high-resolution output image 192 havinga resolution which is higher than the original by performingsuper-resolution interpolation on the low-resolution output image.

FIG. 4B is a block diagram of a conventional image acquisition systemsimilar to the system shown in FIG. 4A but with an additionalanti-aliasing filter 110. The system of FIG. 4B is similar to that ofFIG. 4A except that the input images are filtered by an anti-aliasingfilter 110 before sampling. Hence, the image quality of thelow-resolution output images 191 is enhanced since folding noise isremoved. However, due to the anti-aliasing filtering, is thehigh-resolution images 192 which are outputted by the super-resolutioninterpolation unit 160 do not contain any finer image details than inthe low-resolution images.

Super resolution is described with reference to FIG. 5. A conventionalsuper-resolution reconstruction method basically includes two steps. Ina first step, motion estimation and registration are performed on theinput images. Motion estimation is to estimate a motion in eachreference image with respect to corresponding current low-resolutionimage with sub-pixel precision. Registration is to register referenceimages on a high-resolution grid 520 corresponding to the currentlow-resolution image using the estimated motion.

In the second step, nonuniform interpolation techniques can be employedto obtain interpolated values for each point of the high-resolution grid520 so as to produce a reconstructed high-resolution image 530. This isdisclosed in, for example, Patent Reference 1.

-   Patent Reference 1: Japanese Laid-open Patent Application    Publication No. 2000-339450.

SUMMARY OF THE INVENTION

However, according to the conventional technique, the system (FIG. 4B)which includes the anti-aliasing filter 110 entails a problem that thedefinition of the super-resolution image is decreased, although thesystem prevents aliasing and makes it easy to perform registration. Incontrast, the system (FIG. 4A) which does not include the anti-aliasingfilter 110 entails a problem that aliasing occurs even when thedefinition of the super-resolution image is increased, although thesystem makes it difficult to perform registration. In other words, thepresence of the anti-aliasing filter 110 illuminates that prevention ofaliasing and simplification of registration are in a trade-offrelationship with increase in the definition of a super-resolutionimage.

This problem is described below with reference to the drawings.

FIGS. 6A to 6C show effects of super-resolution interpolation in spatialand frequency domains. FIG. 6A shows an under-sampled signal f·g and thecorresponding spectrum F*G with aliasing. FIG. 6B illustrates the resultof increasing the resolution of the under-sampled signal by means of aconventional interpolation method. Due to the sampling theorem, theunder-sampled signal does not contain any information regarding videofrequencies which is higher than the Nyquist frequency ω_(N). Hence,although the periodicity in the spectrum is reduced due to theup-sampling, the spectrum is zero for frequencies between the Nyquistfrequency ω_(N) and its alias frequency (shown as a solid line on theright-hand side of FIG. 6B).

Super-resolution interpolation, however, can exploit the folding noise610 in the up-sampled signal in order to reconstruct video frequencies620 which is higher than the Nyquist frequency ω_(N) of the samplingunit. As shown in FIG. 6C, the thus reconstructed signal resembles asignal that would have been generated by sampling the original signal atthe higher resolution in crest portions.

Super-resolution interpolation is not in contradiction to the SamplingTheorem because the additional information is extracted fromlow-resolution images with sub-pixel shifts as described above.Summarizing, super-resolution interpolation can sort out aliasingcomponents in the frequency domain and fill the gap exceeding theNyquist frequency so as to reconstruct image details at a resolutionthat are not apparent from any of the low resolution images taken foritself. Hence, folding noise is important for super-resolutioninterpolation.

On the other hand, folding noise may severely hamper motion estimation.A well known example of this problem is a carriage's spoke wheels, whichappear to rotate in the wrong direction in a Western film. Although thisadverse effect is rather due to aliasing in the temporal domain, thesame problem arises with under-sampling in the spatial domain.Therefore, it may be necessary to employ anti-aliasing filtering inorder to be able to perform motion estimation, which is a prerequisitefor super-resolution interpolation.

FIG. 7A illustrates the effect of employing an anti-aliasing filter asshown in FIG. 1B. FIG. 7A exhibits the anti-aliasing filtered andsampled signal of FIG. 3E. Up-sampling this signal, i.e., addingadditional sampling points r, leads to the spectrum shown on theright-hand side of FIG. 7B. Similar to the previous example of FIG. 6B,the periodicity of the signal in the frequency domain is reduced leavinga gap exceeding the Nyquist frequency ω_(N) and its alias. In contrastto the previous example, however, high video frequencies have beenremoved by the anti-aliasing filter and thus have not been folded overto produce aliasing components. Hence, information related to thesefrequencies is irrevocably lost and cannot be reconstructed.

Digital image acquisition systems usually suffer from a limitation ofthe resolution provided by the imaging device. Apart from practicalrestrictions with respect to details that can or cannot be discerned inthe sampled image, the limited sampling resolution can also lead todisturbing artifacts, such as Moiré patterns. These artifacts are aconsequence of sampling an original image containing fine patterns witha resolution that is not high enough to faithfully reproduce thesepatterns. This is an example of the well-known effect of aliasing inundersampled signals. Therefore, conventional digital image acquisitionsystems apply an anti-aliasing filter before sampling the image in orderto prevent these disturbing artifacts. The anti-aliasing filterbasically blurs the original image so as to remove those patterns thatare too fine to be sampled anyway and thus prevents formation of Moirépatterns. In this conventional approach, however, details removed by theanti-aliasing filter are permanently lost and cannot be reconstructed bysuper-resolution techniques.

The present invention has an object of providing an image processingapparatus, an image processing method, a program, and a semiconductorintegrated circuit for achieving both (i) prevention of aliasing andsimplification of registration and (ii) enhancement of the definition ofa super-resolution image.

In order to solve the above problem, the image processing apparatus ofthe present invention includes a filter unit which filters imagesignals; a sampling unit which generates first digital image signalshaving a first resolution by sampling the filtered image signals at apredetermined sampling frequency; and a super-resolution unit whichreconstructs a second digital image signal having a second resolutionwhich is higher than the first resolution by performing super-resolutioninterpolation on the first digital image signals generated by thesampling unit, wherein the filter unit passes frequency componentscorresponding to or lower than the Nyquist frequency which is half thesampling frequency, and passes a part of frequency components within arange from the Nyquist frequency to the highest frequency which can berepresented by the second resolution. As described above, the presentinvention takes a unique scheme of leaving, under control, a part ofvideo frequencies exceeding the Nyquist frequency of a samplingresolution. This makes it possible to achieve both (i) prevention ofaliasing and simplification of registration and (ii) enhancement of thedefinition of a super-resolution image.

Here, the image processing apparatus may further include an inversefilter unit which has a filter characteristic which is inverse to afilter characteristic of the filter unit and to filter the seconddigital image signal.

Here, the inverse filter unit may attenuate the frequency componentscorresponding to or lower than the Nyquist frequency, and pass the partof frequency components within the range from the Nyquist frequency tothe highest frequency which can be represented by the second resolution.

Here, the inverse filter unit may pass the frequency componentscorresponding to or lower than the Nyquist frequency, and emphasizefrequency components corresponding to or higher than the Nyquistfrequency.

The inverse filter unit structured like this can enlarge high-frequencycomponents in digital images having the second resolution attenuated bythe filter unit, in other words, can obtain an image signal having thesecond resolution representing sharper image details.

Preferably, filter characteristics of aliasing control filters areadaptively set depending on the content of an input image. Theattenuation coefficient optimum for an aliasing control filter dependson the content of the input image. In other words, it depends on theamount of fine details in video frequencies close to the Nyquistfrequency. Leaving information for performing super-resolutioninterpolation as much as possible using aliasing control filters for thecontent of the input image leads to a reduction in the amount of foldingnoise in low-resolution digital images.

Here, the filter unit may have a filter characteristic of having anattenuation slope in the range from the Nyquist frequency to the highestfrequency, and have a filter characteristic of attenuating, to 0, afrequency component corresponding to the highest frequency.

Here, it is preferable that digital image signals having the firstresolution correspond to a sequence of frames. In this case, thelow-resolution first image signals can be easily recorded using a videocamera. Further, the video sequence recorded by the video cameracontains sub-pixel shifts between the frames. This is basic preparationfor performing super-resolution.

Here, the super-resolution unit may reconstruct high-resolution digitalimages. The high-resolution second digital images correspond tohigh-resolution digital images reconstructed from the video sequence. Inthis way, it becomes possible to record a low-resolution video sequenceand generate a high-resolution version having the same content.

In addition, the image processing method, the program and thesemiconductor integrated circuit according to the present invention havethe same structure as described above.

With the image processing apparatus according to the present invention,it becomes possible to achieve both (i) prevention of aliasing andsimplification of registration and (ii) enhancement of the definition ofa super-resolution image. Further, it becomes possible to enlargehigh-frequency components in digital images having high-resolution (thesecond resolution) through super resolution, in other words, to obtain ahigh-resolution image signal representing sharper image details.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1A is a block diagram showing the structure of a conventional imageacquisition system.

FIG. 1B is a block diagram showing the structure of a conventional imageacquisition system with anti-aliasing filtering.

FIG. 2A is a diagram of an input signal in a spatial domain and afrequency domain.

FIG. 2B is a diagram of a Dirac comb, as it is used in the samplingstep, in the spatial domain and the frequency domain.

FIG. 2C is a diagram of the sampled input signal in the spatial domainand the frequency domain.

FIG. 3A is a diagram of an input signal in a spatial domain and afrequency domain.

FIG. 3B is a diagram of a low pass filter in the spatial domain and thefrequency domain.

FIG. 3C is a diagram of the low-pass filtered input signal in thespatial domain and the frequency domain.

FIG. 3D is a diagram of a Dirac comb, as it is used in the samplingstep, in the spatial domain and the frequency domain.

FIG. 3E is a diagram of the low-pass filtered input signal in thespatial domain and the frequency domain.

FIG. 4A is a block diagram of a conventional image acquisition systemwith super-resolution interpolation.

FIG. 4B is a block diagram showing the structure of a conventional imageacquisition system with super-resolution interpolation and anti-aliasingfiltering.

FIG. 5 is an illustration of the registration and interpolation insuper-resolution reconstruction.

FIG. 6A is a diagram of a sampled input signal in a spatial domain and afrequency domain.

FIG. 6B is a diagram of a resolution-enhanced signal in the spatialdomain and the frequency domain.

FIG. 6C is a diagram of a resolution-enhanced signal in the spatial andthe frequency domain using super-resolution interpolation.

FIG. 7A is a diagram of an anti-aliasing filtered and sampled inputsignal in a spatial domain and a frequency domain.

FIG. 7B is a diagram of super-resolution interpolation applied to ananti-aliasing filtered and sampled input signal in the spatial domainand the frequency domain.

FIG. 8 is a block diagram showing the structure of an image processingapparatus according to an embodiment of the present invention.

FIG. 9A is a diagram of an input signal in a spatial domain and afrequency domain.

FIG. 9B is a diagram of a low pass filter in the spatial domain and thefrequency domain according to the embodiment of the present invention.

FIG. 9C is a diagram of the low-pass filtered input signal in thespatial domain and the frequency domain according to the embodiment ofthe present invention.

FIG. 10 is a diagram illustrating an example of the structure of ade-attenuation filter.

FIG. 11A shows the filtering characteristics of an exemplary aliasingcontrol filter.

FIG. 11B shows the filtering characteristics of an exemplary aliasingcontrol filter.

FIG. 11C shows the filtering characteristics of an exemplary aliasingcontrol filter.

FIG. 12A shows the filtering characteristics of an exemplaryde-attenuation filter.

FIG. 12B shows the filtering characteristics of an exemplaryde-attenuation filter.

FIG. 12C shows the filtering characteristics of an exemplaryde-attenuation filter.

FIG. 13A is a diagram of the low-pass filtered and sampled input signalin a spatial domain and a frequency domain according to the embodimentof the present invention.

FIG. 13B is a diagram of super-resolution interpolation performed in thespatial domain and the frequency domain according to the embodiment ofthe present invention.

FIG. 13C is a diagram of the de-attenuation filtered andsuper-resolution interpolated input signal in the spatial domain and thefrequency domain according to the embodiment of the present invention.

FIG. 14A illustrates aliasing control and the filter characteristics ofa de-attenuation filter having an attenuation coefficient of 0.4according to the embodiment of the present invention.

FIG. 14B illustrates aliasing control and the filter characteristics ofa de-attenuation filter having an attenuation coefficient of 0.2according to the embodiment of the present invention.

FIG. 15 is a block diagram showing the structure of a hierarchical videoencoder according to the embodiment of the present invention.

FIG. 16 is a block diagram showing the structure of a hierarchical videodecoder according to the embodiment of the present invention.

NUMERICAL REFERENCES

-   -   101 input image    -   110 anti-aliasing filter    -   120 sampling unit    -   150 processing/recording unit    -   160 super-resolution interpolation unit    -   190, 191 low-resolution output image    -   192 high-resolution output image    -   520 grid    -   530 high-resolution image    -   610 noise    -   620 video frequency    -   1001 input image    -   1002 high-resolution input image    -   1010 aliasing control filter    -   1020 sampling unit    -   1021 down-sampling unit    -   1051, 1052, 1053 coding unit    -   1054, 1055 adder    -   1056, 1057 decoder    -   1060 super-resolution interpolation unit    -   1070 de-attenuation filter    -   1091, 1098 low-resolution output image    -   1092, 1099 high-resolution output image    -   1095, 1096 bitstream

DETAILED DESCRIPTION OF THE INVENTION

An image processing apparatus in an embodiment according to the presentinvention includes a filter unit which filters image signals; a samplingunit which generates first digital image signals having a firstresolution by sampling the filtered image signals at a predeterminedsampling frequency; and a super-resolution unit which reconstructs asecond digital image signal having a second resolution which is higherthan the first resolution by performing super-resolution on the firstdigital image signals generated by the sampling unit.

Here, the filter unit functions as an anti-alias control filter, not asa simple anti-alias filter. The filter unit as the anti-alias controlfilter passes frequency components corresponding to or lower than theNyquist frequency which is half the sampling frequency, and passes apart of frequency components within a range from the Nyquist frequencyto the highest frequency which is can be represented by the secondresolution. In other words, the filter unit does not remove all of thefrequencies exceeding the Nyquist frequency, and passes a part offrequency components within a range from the Nyquist frequency to ahighest frequency which can be represented by the second resolution. Thefrequency components within the range from the Nyquist frequency to thehighest frequency are essential to enhance the definition by performingsuper resolution although it causes aliasing. The video frequenciesexceeding the Nyquist frequency is not removed but attenuated in orderto reduce disturbing folding noise in the low-resolution output image1091. The definition of the image sampled and recorded in this way canbe further enhanced by the super-resolution interpolation unit 1060 forgenerating a high-resolution output image having a resolution which ishigher than the sampling resolution. In addition, since the frequencycomponents causing aliasing are attenuated, it is possible to increaseaccuracy in registration without deteriorating the accuracy in motionestimation in super resolution.

In addition, the image processing apparatus further has inverse filtercharacteristics. The inverse filter unit filters a second digitalsignal, and has filter characteristics which are inverse to the filtercharacteristics of the filter unit. Hence, the inverse filter unit canenlarge or emphasize high-frequency components in the second digitalimage with the second resolution attenuated by the filter unit. In otherwords, an image signal having the second resolution representing sharperimage details can be generated.

FIG. 8 is a block diagram showing an exemplary structure of the imageprocessing apparatus in an embodiment according to the presentinvention. The image processing apparatus in the drawing includes analiasing control filter 1010, a sampling unit 1020, aprocessing/recording unit 1050, a super-resolution interpolation unit1060, and an inverse attenuation filter 1070.

The input images 1001 are inputted to an aliasing control filter 1010before sampling. Input images may either be analogue images produced byan optical system or high-resolution digital images transmitted fromanother imaging device and the like.

In order to allow for super-resolution interpolation of the sampledimages, the aliasing control filter 1010, which corresponds to thefilter unit, does not remove all the frequencies exceeding the Nyquistfrequency of the sampling unit 1020. Instead, video frequencies abovethe Nyquist frequency are attenuated in order to reduce disturbingfolding noise in the low-resolution output images 1091.

For example, the aliasing control filter 1010 may be mounted in form ofan optical (blurring) filter used in the upstream of an image sensor,and may be implemented in form of a digital filter which operates ondigital image information having an input resolution which is higherthan the sampling resolution. In the former case, the sampling unit 1020may be a digital image sensor such as a CCD. In the latter case, thesampling unit 1020 may be the same as the aliasing control filter 1010in a sense that the aliasing control filter 1010 generates an image byfiltering and down-sampling the input image 1001.

The sampling unit 1020 generates digital image signals having a firstresolution (referred to as a low resolution hereinafter) by sampling thefiltered image signal at a predetermined sampling frequency.

The processing/recording unit 1050 outputs, as a low-resolution outputimage 1091, a low-resolution digital image from the sampling unit 1020,and outputs it to the super-resolution interpolation unit 1060. Inaddition, the process/recording unit 1050 may further process thedigital image or record it onto a recording medium.

The super-resolution interpolation unit 1060 reconstructs a digitalimage signal having a second resolution (referred to as a highresolution hereinafter) which is higher than a first resolution byperforming super resolution on low-resolution digital image signalsoutputted by the processing/recording unit 1030.

The de-attenuation filter 1070 functions as the inverse filter unit, andfilters the high-resolution image from the super resolutioninterpolation unit 1060.

Moreover, the low-resolution output images 1091 and/or the reconstructedhigh-resolution output images 1092 may be outputted by means of anoutput unit. The output unit may be connected to a display device fordisplaying the output images, a storage device for storing the outputimages, an encoder/transmitter for encoding and transmitting the outputimages over a communications line, or an image processing unit such asan image recognition or motion estimation unit, etc.

Whether only low-resolution, only high-resolution, or both low- andhigh-resolution output images are outputted via the output unit maydepend on requirements and capabilities of a down-stream device, such asdisplay resolution, storage capacity, transmission bandwidth,computational performance, etc.

Next, operations of the image processing apparatus are described indetail with reference to the drawings.

Functions of the aliasing control filter 1010 are described withreference to FIGS. 9A to 9C. For simplification, signals are assumed tobe one-dimensional signals here. In each of the drawings, the left-handside graph shows a signal in the spatial domain. The horizontal axis xshows a one-dimensional spatial coordinate or a time axis, and thelongitudinal axis shows luminance. On the other hand, the right-handside graph shows a signal transformed (Fourier transformed) in thefrequency domain. The horizontal axis ω shows frequency (radian), thelongitudinal axis shows the strength of frequency components of thesignal, and ω_(N) shows the Nyquist frequency.

FIG. 9A shows an input signal f. FIG. 9B shows a characteristic h_(Ac)of the aliasing control filter 1010. FIG. 9C shows f*h_(Ac) which is theresult of filtering the input image 1001 using the aliasing controlfilter 1010.

Note that the aliasing control filter 1010 does not totally removefrequencies exceeding the Nyquist frequency E⁾N in the sampling unit1020. Instead, high image frequencies are maintained (no attenuation) atthe Nyquist frequency, and attenuated by a frequency-dependentattenuation coefficient down to zero (removal) at a frequencycorresponding to the highest image frequency that can be represented bythe resolution of the super-resolution interpolation unit. Consequently,the impulse response h_(AC) of this filter in the spatial domain isnarrower than the impulse response h_(AA) of the anti-aliasing filtershown in FIG. 3B. The input signal filtered by the aliasing controlfilter 1010 thus still contains image frequencies exceeding the Nyquistfrequency (cf. FIG. 9C).

FIG. 13A shows the result of sampling input images filtered by thealiasing control filter 1010. FIG. 13A is analogous to FIG. 3E, butdifferent in that a part of image frequencies exceeding the Nyquistfrequency is left. Because of the presence of image frequencies which ishigher than the Nyquist frequency, the spectra of the filtered inputsignal overlap with each other, and weak aliasing occurs (cf. the shadedareas 310 and 320 in FIG. 13A). Due to the aliasing control filter 1010,however, the amplitude of the folding noise is reduced accordingly.Hence, the thus generated low-resolution images exhibit an improvedimage quality as compared to the low-resolution images outputted by theconventional image acquisition system shown in FIG. 4A.

Moreover, due to the suppression of folding noise, motion can beestimated more accurately at a sub-pixel level, which is required forthe registration of super-resolution interpolation. The remainingfolding noise can be exploited by the super-resolution interpolationunit 1060 to extract image details at a resolution superior to thesampling resolution. Ideally, aliasing in the frequency domain can becompletely resolved leading to the reconstructed signal shown as a solidline in FIG. 13B.

However, the reconstructed signal still suffers from an attenuation ofhigh frequency components introduced by the aliasing control filter 1010(the shaded area 330). This is corrected by the de-attenuation filter1070 that amplifies those image frequencies so as to compensate theeffect of the aliasing control filter 1010. As a result, the originalinput signal can be reconstructed at a high resolution (cf. FIG. 13C).

In this description, the de-attenuation filter 1070 is described asbeing a separate device accepting the output of the super-resolutioninterpolation unit 1060. However, the present invention is notrestricted in this respect. Alternatively, the de-attenuation filter mayas well be a part of the super-resolution interpolation unit 1060, inparticular a part of its observation model for generating the observedlow-resolution images from the original input signal.

The characteristics of the aliasing control filter 1010 shown in FIG. 9Bis a mere example, and not supposed to restrict the present invention.As shown in FIGS. 11A to 11C, the aliasing control filter 1010 capableof switching among various types of filter characteristics may beemployed in order to achieve the aim of the present invention.Preferably, the filter has a constant gain up to the Nyquist frequency,and a certain attenuation rate exceeding the frequency. The optimumdamping factor for frequencies exceeding the Nyquist frequency maydepend on the image content. The filter characteristics of FIGS. 11A to11C may be adaptively switched. Preferably, such attenuation rate is setthat leaves high-frequency components strong enough to prevent foldingnoise. Otherwise, attenuation rate is preferably as weak as possible,i.e., the attenuation rate should be a value equal to or less than 1, inorder to leverage super-resolution interpolation by the super-resolutioninterpolation unit 1060.

Experiments have shown that the attenuation rate for frequenciesexceeding the Nyquist frequency is preferably about 0.1 to 0.5.

The aliasing control filter 1010 may be a filter which samples analogimage signals, or may be a filter which down-samples digital imagesignals. FIG. 10 is a diagram of an exemplary structure of the aliasingcontrol filter 1010 corresponding to the latter. The aliasing controlfilter 1010 in the drawing includes (2N+1) multipliers and an adder.Inputted into the (2N+1) multipliers are consecutive (2N+1) pixels(sample) P_(−N) to P_(N) and (2N+1) weight coefficients (tapcoefficients) W_(−N) to W_(N). The filter characteristics are determineddepending on how such weight coefficients are combined. The adder addsthe results of the (2N+1) multiplications, and outputs a new pixel P′corresponding to P₀. A set of new pixels P′ forms a low-resolutionimage.

The de-attenuation filter 1070 has filter characteristics which areinverse to the filter characteristics of the aliasing control filter1010. Thus, the de-attenuation filter 1070 has a single gain up to theNyquist frequency, and a certain emphasis characteristic for frequenciesexceeding the Nyquist frequency. Preferably, the gain in a frequencyexceeding the Nyquist frequency is between 2 to which corresponds to theattenuation coefficient of the aliasing control filter 1010.

Each of FIGS. 12A to 12C shows the characteristics, of a de-attenuationfilter 1070, which are inverse to the filter characteristics in acorresponding one of FIGS. 11A to 11C. The de-attenuation filter 1070can switch among such filter characteristics working with the aliasingcontrol filter 1010, pass frequency components less than the Nyquistfrequency (gain 1), and emphasize the frequency components greater thanthe Nyquist frequency (gains 2 to 10). The de-attenuation filter 1070can be structured in form of a circuit as shown in FIG. 10. The filtercharacteristics which are inverse to the filter characteristics of thealiasing control filter 1010 are determined depending on how theseweight coefficients are combined.

Note that the de-attenuation filter 1070 may be structured to attenuatefrequency components less than the Nyquist frequency (for example, aconstant gain of 0.5) and pass a part of frequency components within arange from the Nyquist frequency to the highest frequency which can berepresented by the second resolution (for example, gain 1).

Technically, both the aliasing control filter 1010 and thede-attenuation filter 1070 are implemented preferably as finite impulseresponse filters. FIGS. 14A and 14B show two examples of 9-tap aliasingcontrol filters 1010 with attenuation rates 0.4 (101 a in FIG. 14A) and0.2 (1101 b in FIG. 14B), respectively, as well as the correspondingideal de-attenuation filters (1102 a, 1102 b) and their finite (9-tap or11-tap) implementations (shown as dashed lines in FIGS. 14A and 14B).

A description is given of the image processing apparatus, focusing on acurrent image among low-resolution images. So far, the present inventionhas been described in terms of acquiring, processing, and reproducingsingle images. However, the present invention is not restricted tosingle-image applications, but rather may also be applied toacquisition, processing and reproduction of sequences of images, i.e.,to video applications, or is to multiview images, i.e., to images orvideos recorded simultaneously from the same scene but from differentangles.

Next, a preferred embodiment of the present invention in the context ofvideo coding apparatus is described next, with reference to FIGS. 15 and16.

FIG. 15 is a block diagram of the hierarchical structure of a videocoding apparatus in the present invention. The digital high-resolutioninput image 1002 is filtered by the aliasing control filter 1010, andcontrols the amount of folding noise generated by the next down-samplingunit 1021. The down-sampling unit 1021 outputs low-resolution digitalvideo signals by down-sampling the filtered input video signals. Thedown-sampled video signals can be applied to the display of a mobiledevice having a limited display capability. The coding unit 1051 codesthe down-sampled video signals, and outputs, as bitstream 1 (1095),resulting compressed digital video signals. The coding unit 1051 may bea video encoder for transmission in accordance with the MPEG-2 or H.264/MPEG4-AVC Standards. The present invention, however, is not limitedto the specific type of encoder. Rather, the present invention can beapplied to any type of encoder which codes a video signal into a digitalbitstream.

The bitstream 1 is sent to an internal coding unit 1053 to generate areference video signal. This reference video signal is sent to thesuper-resolution interpolation unit 1060 in order to reconstruct ahigh-resolution video signal from a compressed low-resolution bitstream1. The high-resolution video signal generated from the super-resolutioninterpolation 1060 is filtered by the de-attenuation filter 1070.

Preferably, the resolution of the reconstructed signal corresponds tothe resolution of the high-resolution input image 1002.

The use of super-resolution interpolation technique is disables thesuper-resolution interpolation unit 1060 to execute up-sampling on aframe-by-frame basis.

Instead, the reconstruction is based on images taken from a sequence ofconsecutive frames. Nevertheless, the reconstruction can be executed ona macro block level, i.e., the images may correspond to macro blocksfrom consecutive frames. Further, the super-resolution interpolation mayexploit motion vectors estimated by a motion estimation/compensationunit that is generally a part of a video encoder. Preferably, the motionvector (MV) estimated by a first encoding unit 1051 or a second encodingunit 1052 may be used for the super-resolution interpolation.Alternatively, more precise motion vectors may be estimated by thesuper-resolution interpolation unit 1060.

An adder 1054 subtracts the reconstructed signal from thehigh-resolution input image 1002 in order to generate a differencesignal containing the image information that could not be reconstructedfrom the compressed low-resolution video signal. The difference signalis fed to the second coding unit 1052 of a similar kind as the firstcoding unit 1051 in order to code the difference signal into a bitstream2 (1096).

The high-resolution input image 1002 has thus been coded into thefollowing compressed video signals: the bitstream 1 containing videoinformation at a reduced resolution; the bitstream 2 containing thedifferences between the original video signal and a video signalreconstructed from the low-resolution signal by means ofsuper-resolution interpolation and de-attenuation. The bitstream 1 isself-contained in the sense that it can be decoded independently, forinstance, to mobile communication devices with limited displaying and/orcomputing capabilities. For this purpose, image quality can be optimizedby means of the aliasing control filter 1010 that reduces the amount offolding noise.

Both bitstreams 1 and 2 may be multiplexed into a single bitstream thusforming a hierarchically coded compressed video signal corresponding tothe high-resolution input image 1002. The multiplexed bitstreams may betransmitted or recorded irrespective of the display capabilities or adecoder. The multiplexed bitstreams represent the input signals at fullresolution. Due to the super-resolution interpolation and de-attenuationprocessing, however, redundancies related to the high resolution havebeen eliminated thus enabling a high compression ratio without adverselyaffecting image quality.

The aliasing control filter 1010 thus allows optimization of the imagequality of the low-resolution version of the video data on the one hand,and the overall coding efficiency of the full-resolution video data onthe other hand. The stronger the attenuation of image frequenciesexceeding the Nyquist frequency of the down-sampling unit 1021, thebetter the image quality of the low-resolution version since foldingnoise is suppressed. However, a too little amount of aliasing componentsimpairs super-resolution interpolation, and thus deteriorates theprediction of the high-resolution video data from the low-resolutionversion. This leads to a larger difference signal at the adder 1054, andthus to more data that has to be coded in the bitstream 2.

FIG. 16 is a block diagram showing the structure of a video decodingapparatus according to the present invention. The bitstreams 1 and 2(1095, 1096) outputted by the video coding apparatus are fed intorespective decoding units 1056 and 1057. The low-resolution output image1098 of the first decoding unit 1056 thus corresponds to thelow-resolution version of the coded high-resolution input image 1002.The output of the first decoding unit 1056 is also fed into asuper-resolution interpolation unit 1060 followed by a de-attenuationfilter 1070 in order to reconstruct a high-resolution video signal fromthe low-resolution version. Preferably, the motion vector information(MV) required for decoding the bitstream 1 is also fed to thesuper-resolution interpolation unit 1060. Alternatively, more precisemotion vectors may be estimated by the super-resolution interpolationunit 1060 itself. This motion vector may be exploited for registeringthe low-resolution images to the high-resolution grid as described inconjunction with FIG. 5. The output of the decoding unit 1057 representsthe difference signals, and is added to the high-resolution videosignals reconstructed by the adder 1055. This yields the decompressedhigh-resolution output image 1099 corresponding to the codedhigh-resolution input image 1002.

Depending on the displaying and computing capabilities of the decodingdevice, only the decoding unit 1056 may be mounted, and only thelow-resolution images may be decoded. This achieves a reduction of costfor production and operation of the devices as well as in a reduction ofcircuit scale, etc.

On the other hand, the very same multiplexed bitstream may be decoded byhigh-quality decoding devices so as to reproduce the video signal atfull resolution. Due to the super-resolution and de-attenuationprocessing, redundancies within the video data have been significantlyreduced thus leading to an improved coding efficiency. The high-qualitydecoding device may thus reproduce the video signal by receiving andprocessing a fewer amount of encoded data than decoding devicesoperating with conventional video coding schemes.

As explained above, the optimum filter characteristics of the aliasingcontrol filter 1010 may depend on the image content. Hence, it isadvantageous to adapt the filtering characteristics of the aliasingcontrol filter 1010 to the image content of the high-resolution inputimage 1002 in order to ensure that visible folding noise in thedown-sampled video images are suppressed, while sufficient frequencycomponents exceeding the Nyquist frequency remains available forsuper-resolution interpolation. Consequently, the filteringcharacteristics of the de-attenuation filter 1070 are preferably adaptedas well. In a preferred embodiment, the video image is hence analyzed byan image analyzer (not shown) for setting the filter characteristics ofthe aliasing-control filter 1010 and the de-attenuation filter 1070 inreal-time.

The optimum attenuation coefficient of the aliasing control filter 1010may, for instance, depend on the amount of details contained within theinput video. Especially, those details with image frequencies close tothe Nyquist frequency are particularly sensitive to folding noise. Theattenuation coefficient of the aliasing control filter 1010 may thus becontrolled in accordance with the spectral power of the input videosignal in a frequency range close to the Nyquist frequency.

In the setup of video coding apparatus and video decoding apparatusshown in FIGS. 15 and 16, the filtering characteristics that is employedby the de-attenuation filter 1070 in the video encoding apparatus may besignaled to the decoder. The filtering characteristics of thede-attenuation filter 1070 of the video decoding apparatus can thus beadapted accordingly in order to reproduce the high-resolution outputimage 1099.

Signaling of the filtering characteristics is performed by insertingsignaling information in the bitstream 1096. The signaling informationmay comprise a full definition of the filter, e.g. in form of filtercoefficients such as those of a finite impulse response filter, orcertain parameters such as attenuation coefficient and thresholdfrequency.

The video coding apparatus according to the present invention includesan aliasing control filter 1010 which controls folding noise rather thanremoving it, and thereby achieving an enhanced image quality of asuper-resolution image according to super-resolution interpolationtechnique. A preferred embodiment of the present invention relates tohierarchical video data compression and decompression with improvedcoding efficiency. The video data is coded into two bitstreams. Thebitstream 1 is a self-contained representation of a low-resolutionversion of the video data. The bitstream 2 only contains the differencebetween the full-resolution video data and its super-resolutionreconstruction.

As shown above, the image coding apparatus in the embodiment includes ade-attenuation filter 1070 having filter characteristics which areinverse to the filter characteristics of the aliasing control filter1010 for enlarging video frequency of a high-resolution video signalattenuated by the aliasing control filter 1010. The adder 1054 subtractsthe output of the de-attenuation filter from the originalhigh-resolution input image, and outputs a difference signal, so thatthe subtracter 1054 outputs the difference. This increases thereproducibility of high-frequency components, yielding sharper images.In this way, the data amount which has to be coded in the bitstream 2 isdecreased to an improved overall coding efficiency.

The filter characteristics of the aliasing control filter 1010 may beadaptively switched depending on the content of the high-resolutioninput image 15. This makes it possible to leave information forsuper-resolution interpolation as much as possible, and to suppressfolding noise by means of a low-resolution digital video signal.

Preferably, the filter characteristics information of an aliasingcontrol filter is signaled to the video decoding apparatus. It is onlynecessary that the filter characteristics information is inserted to thebitstream 2. The filter characteristics information may be a list offilter coefficients, and may include a limit frequency and anattenuation coefficient. By acquiring the characteristics of thede-attenuation filter 1070, the video decoding apparatus can decodehigh-resolution digital video signals.

Motion vector information estimated by the video decoding apparatus maybe sent, as an input, to the super-resolution reconstruction unit, inorder that the high-resolution video signal is reconstructed.Super-resolution interpolation requires information of sub-pixel motionsin consecutive images. The information is extracted from motion vectorinformation determined in advance in motion compensation. Thecalculation efficiency is improved in this way.

The image coding apparatus may further include a bitstream multiplexerwhich multiplexes the bitstream 1 and bitstream 2 into an outputbitstream which displays an input video signal. In this way, the codedvideo data can be easily transmitted via a single communication path, orcan be recorded onto a recording medium.

Preferably, at least one of these two coding units codes an input signalinto a bitstream in accordance with the video compression standard. Suchvideo encoder (for example, an MPEG-II or H.264/AVC encoder) is animplementation of an advanced technology, and provides the optimumperformances and coding efficiency.

In addition, the video decoding apparatus in this embodiment decodesvideo data coded by the image coding apparatus which uses suchde-attenuation filter 1070 in order to reconstruct high-resolutionreference video signals. In this way, it is possible to achieve a highcoding efficiency by efficiently removing redundancies in a currentvideo signal to be coded.

Preferably, the filter characteristics of the de-attenuation filter isset according to the filter characteristics information signaled from anencoder. The filter characteristics information may extract thebitstream 2. The filter characteristics information may be a list offilter coefficients. The filter characteristics information may be alimit frequency and a de-attenuation coefficient. The video decodingapparatus can is adapt the de-attenuation in order to optimize thequality of the high-resolution digital video signals in this way.

Preferably, the motion vector information (MV) found by the decodingunit 1056 may be sent, as an input, to the super-resolutioninterpolation unit 160 in order that the high-resolution video signal isreconstructed. Super-resolution interpolation requires information ofsub-pixel motions in consecutive images. The information is extractedfrom the motion vector information determined in motion compensation.The calculation efficiency is improved in this way.

Preferably, a bitstream de-multiplexer is provided. The bitstreamde-multiplexer is user for de-multiplexing the bitstream 1 and thebitstream 2 from the input bitstream. In this way, it is easy to decodecoded video data transmitted via a single transmission channel orrecorded on a recording medium.

Preferably, at least one of these two decoding units decodes the inputbitstream in accordance with the video compression standard. Note thateach block diagram shown in the embodiment is typically implemented asthe LSI that is an integrated circuit device. This LSI may beimplemented on a single chip or on several chips. A block called as anLSI here may be called as an IC, a system LSI, a super LSI or an ultraLSI depending on the integration degree.

An integrated circuit is not necessarily implemented in a form of anLSI, it may be implemented in a form of an exclusive circuit or ageneral purpose processor. It is also possible to use the FieldProgrammable Gate Array (FPGA) that enables programming or areconfigurable processor that can reconfigure the connection or settingof a circuit cell inside the LSI after generating an LSI.

Further, in the case where technique of implementing an integratedcircuit that supersedes the LSI is invented along with the developmentin semiconductor technique or another derivative technique, as a matterof course, integration of the function blocks may be implemented usingthe invented technique. Bio technique is likely to be adapted.

Note that the main part may also be implemented by a processor or aprogram shown in the respective blocks of block diagrams.

The present invention is applicable to an image processing apparatus, avideo coding apparatus, and a video decoding apparatus, and inparticular applicable to a video recording and reproducing apparatus, avideo camera, and a television camera.

1. An image processing apparatus, comprising, a filter unit configuredto filter image signals; a sampling unit configured to generate firstdigital image signals having a first resolution by sampling the filteredimage signals at a predetermined sampling frequency; and asuper-resolution unit configured to reconstruct a second digital imagesignal having a second resolution which is higher than the firstresolution by performing super-resolution interpolation on the firstdigital image signals generated by the sampling unit, wherein the filterunit is configured to pass frequency components corresponding to orlower than a Nyquist frequency which is half the sampling frequency, andto pass a part of frequency components within a range from the Nyquistfrequency to a highest frequency which can be represented by the secondresolution.
 2. The image processing apparatus according to claim 1,further comprising: an inverse filter unit configured to have a filtercharacteristic which is inverse to a filter characteristic of the filterunit and to filter the second digital image signal.
 3. The imageprocessing apparatus according to claim 2, wherein the inverse filterunit is configured to attenuate the frequency components correspondingto or lower than the Nyquist frequency, and to pass the part offrequency components within the range from the Nyquist frequency to thehighest frequency which can be represented by the second resolution. 4.The image processing apparatus according to claim 2, wherein the inversefilter unit is configured to pass the frequency components correspondingto or lower than the Nyquist frequency, and to emphasize frequencycomponents corresponding to or higher than the Nyquist frequency.
 5. Theimage processing apparatus according to claim 2, wherein the filter unithas a filter characteristic of having an attenuation slope in the rangefrom the Nyquist frequency to the highest frequency, and has a filtercharacteristic of attenuating, to 0, a frequency component correspondingto the highest frequency.
 6. The image processing apparatus according toclaim 2, wherein the filter unit is switchable among filtercharacteristics including a first filter characteristic and a secondfilter characteristic, an attenuation slope starting with the Nyquistfrequency and ending with the highest frequency in the first filtercharacteristic is greater than an attenuation slope starting with theNyquist frequency and ending with the highest frequency in the secondfilter characteristic, and the filter unit is configured to select thefirst filter characteristic in the case where the number ofhigh-frequency components included in the image signal is less than athreshold value, and to select the second filter characteristic in thecase where the number of high-frequency components included in the imagesignal is greater than a threshold value.
 7. The image processingapparatus according to claim 1, wherein the filter unit has a filtercharacteristic of having an attenuation slope in the range from theNyquist frequency to the highest frequency, and has a filtercharacteristic of attenuating, to 0, a frequency component correspondingto the highest frequency.
 8. The image processing apparatus according toclaim 1, wherein the filter unit is switchable among filtercharacteristics including a first filter characteristic and a secondfilter characteristic, an attenuation slope starting with the Nyquistfrequency and ending with the highest frequency in the first filtercharacteristic is greater than an attenuation slope starting with theNyquist frequency and ending with the highest frequency in the secondfilter characteristic, and the filter unit is configured to select thefirst filter characteristic in the case where the number ofhigh-frequency components included in the image signal is less than athreshold value, and to select the second filter characteristic in thecase where the number of high-frequency components included in the imagesignal is greater than a threshold value.
 9. The image processingapparatus according to claim 8, wherein the inverse filter unit isswitchable among a first inverse filter characteristic which is inverseto the first filter characteristic and a second inverse filtercharacteristic which is inverse to the second filter characteristic, andthe inverse filter unit is configured to select one of the first inversefilter characteristic and the second inverse filter characteristic whichcorresponds to the selected one of the first filter characteristic andthe second filter characteristic.
 10. An image processing methodcomprising: filtering image signals; generating first digital imagesignals having a first resolution by sampling the filtered image signalsat a predetermined sampling frequency; and reconstructing a seconddigital image signal having a second resolution which is higher than thefirst resolution by performing super-resolution interpolation on thefirst digital image signals generated by the sampling unit, wherein, inthe filtering, frequency components corresponding to or lower than aNyquist frequency which is half the sampling frequency are passed, and apart of frequency components within a range from the Nyquist frequencyto a highest frequency which can be represented by the second resolutionis passed.
 11. A non-transitory computer-readable recording mediumstoring a program, the program causing a computer to execute stepscomprising: filtering image signals; generating first digital imagesignals having a first resolution by sampling the filtered image signalsat a predetermined sampling frequency; and reconstructing a seconddigital image signal having a second resolution which is higher than thefirst resolution by performing super-resolution interpolation on thefirst digital image signals generated by the sampling unit, wherein, inthe filtering, frequency components corresponding to or lower than aNyquist frequency which is half the sampling frequency are passed, and apart of frequency components within a range from the Nyquist frequencyto a highest frequency which can be represented by the second resolutionis passed.
 12. A semiconductor integrated circuit, comprising: a filterunit configured to filter image signals; a sampling unit configured togenerate first digital image signals having a first resolution bysampling the filtered image signals at a predetermined samplingfrequency; and a super-resolution unit configured to reconstruct asecond digital image signal having a second resolution which is higherthan the first resolution by performing super-resolution interpolationon the first digital image signals generated by the sampling unit,wherein the filter unit is configured to pass frequency componentscorresponding to or lower than a Nyquist frequency which is half thesampling frequency, and to pass a part of frequency components within arange from the Nyquist frequency to a highest frequency which can berepresented by the second resolution.